3D Morphable Model Fitting to Image Sequences

This paper outlines a new method to fit 3D Morphable Models (3DMM’s) from sets of 2D image features. It is the extension of a popular correspondence 3D shape fitting from 2D feature point method which allows it to be applied to video sequences. For shape estimation this paper focuses on strictly linear solutions to the problem of shape fitting to image sequences. In doing so we introduce a ShapeUpdate algorithm which integrates the data present in the video sequence using only a small amount of computation. This paper also presents the relationship between our Shape-Update algorithm and our previous Kalman filter approach. We also describe a pose estimation algorithm that uses an Extended Kalman filter and a linearly estimated prior. 1 3D Morphable Models A 3D Morphable Model (3DMM) is a representation of both the 3D shape and texture of (but not restricted to) the human face. In this paper we focus on 3DMMs for face modeling and extend the popular work of Vetter et al [14, 11]. Vetter introduced the Morphable Model as an approach to extract the threedimensional structure of the human face from a single image using a common analysis-by-synthesis approach. A Morphable Model is built from 3D laser scans of human faces which are then put into dense correspondence [3] using their pixel intensities and 3D shape information. The correspondence of heads is achieved using a modified optical flow algorithm and provides a dense vertex to vertex mapping. Using the corresponding heads, shape and texture matrices are formed, where each column is a vectorized representation of the 3D data. In all cases the dimensionality of each shape and texture matrix is very high and is a prohibitive amount of data to work with. The data must be reduced both for practicality and for model parametrization. This is achieved using Principle Component Analysis (PCA) and this provides the equations for shape and texture variation of the model: ŝ = s̄ + S · diag(σs) · α, t̂ = t̄ + T · diag(σt) · γ (1) where ŝ and t̂ are novel 3N× 1 shape and texture vectors, s̄ and t̄ are the 3N× 1 mean shape and texture vectors, S and T are the 3N×M column (eigenvectors) spaces of the shapes and textures, σ are the corresponding eigenvalues, α and γ are shape and texture coefficients. These linear equations describe the variation of shape and texture within the span of the 3D training data. Naturally by varying the shape and texture coefficients different shapes and textures are formed. An example of the effect of varying the first model coefficient is shown in figure 1. The Morphable Model is very similar to the two-dimensional model called the AAM which was introduced by Cootes et al [5]. Here the addition of a 3D shape makes the problem much more difficult. When fitting, which is the process of aligning the model to the image, a 3DMM must optimize parameters for shape, texture, illumination and rigid body rotation. Vetters approach to fitting −3.5 √ σ1 −2.5 √ σ1 −1.5 √ σ1 s̄ 1.5 √ σ1 2.5 √ σ1 3.5 √ σ1 Figure 1: First mode of variation in 3DMM shape model